Auto-Diagnosis of Field Problems in an Appliance Operating System

The use of network appliances, i.e., computer systems specialized to perform a single function, is becoming increasingly widespread. Network appliances have many advantages over traditional general-purpose systems such as higher performance/cost metrics, easier configuration and lower costs of management. Unfortunately, while the complexity of configuration and management of network appliances in normal usage is much lower than that of general-purpose systems, this is not always true in problem situations. The debugging of configuration and performance problems with appliance computers is a task similar to the debugging of such problems with general-purpose systems, and requires substantial expertise. This paper examines the issues of appliance-like management and performance debugging. We present a number of techniques that enable appliance-like problem diagnosis. These include continuous monitoring for abnormal conditions, diagnosis of configuration problems of network protocols via protocol augmentation, pathbased problem isolation via cross-layer analysis, and automatic configuration change tracking. We also describe the use of these techniques in a problem auto-diagnosis subsystem that we have built for the Data ONTAP operating system. Our experience with this system indicates a significant reduction in the cost of problem debugging and a much simpler user experience.